DeepSeek Beat Me To My Own Idea And I Am Not Okay

I had an idea. A good idea. I called it EMM: External Memory Module. The concept was simple. Train the memory separately. Plug it into the model. Decode vectorized data. O(1) retrieval. Minimal overhead. Elegant.

I wrote notes. I sketched diagrams. I told my team about it. I was going to implement it. I was going to publish it. I was going to be the person who solved the memory problem in transformers.

Then DeepSeek published Engram on January 12, 2026. And I died a little inside.

There is no pain like reading a paper and realizing someone else had your idea first. Especially when they open sourced it before you finished your notes.

What Engram Actually Is

Engram is a conditional external memory module with O(1) constant-time knowledge lookup. It structurally separates static knowledge from the transformer backbone. It uses N-gram hashing to map token sequences to a learnable lookup table. It can store over 100 billion parameters in CPU RAM.

The paper is titled "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models". It was signed by Liang Wenfeng, DeepSeek's founder. It is open source on GitHub. It was developed in collaboration with Peking University.

It is exactly what I wanted to build. Except they built it. And they tested it. And they published it. And they open sourced it.

The Comparison That Hurts

My EMM (Notes Only)

Concept: Train memory separately. Plug into model. Decode vectorized data.

Implementation: None. Still in markdown files.

Results: A folder called EMM_idea/ and growing existential dread.

DeepSeek Engram (Published)

Concept: Conditional memory via scalable lookup. Separate static knowledge from reasoning.

Implementation: Full PyTorch module. GitHub repo. Demo scripts.

Results: Published paper. Open weights. Proven improvements.

They did not just have the idea. They executed. They tested. They published. They open sourced. I have a text file that says "train separately, plug in, decode vectors" and a lot of unused time.

                        # My EMM notes vs. Engram reality

                        Me: "Train it separately. Plug into model. Decode vectorized data."

                        Engram: "Structurally separate static knowledge from transformer backbone."

                        # Same idea. Different levels of completion.

Why This Stings

It is not just that they had the idea first. It is that the core concept is the same. Separate memory from computation. External lookup. Fast retrieval. I wrote this down weeks before January 12. They published on January 12. The timeline hurts.

But here is the thing. Ideas are not unique. Good ideas especially are not unique. Multiple people think of them at similar times. The difference is who ships. DeepSeek shipped. I did not.

Research is not about having ideas. It is about having ideas and then doing the work. I had the idea. I did not do the work.

What I Am Doing About It

I could give up. I could say "they did it better" and move on. I am not doing that. Engram is open source. I can use it. I can learn from it. I can integrate it into my own models.

I am going to use DeepSeek's Engram implementation in TMLM-Sonnet-2. And in Opus. And in future models. Why not? It is open source. It solves the problem I cared about. It is better than what I would have built anyway.

                        # TMLM-Sonnet-2 plans

                        Model: 300M parameters

                        Memory: Engram external module

                        Optimizer: Muon

                        GPU: 5090 OC LC @ 800W

                        ETA: Probably never but we try

                        # Standing on the shoulders of giants.

What I Learned

First, ship faster. If I had started implementing EMM when I first thought of it, maybe I would have something to show. Maybe not better than Engram. But something. Now I have notes and regret.

Second, open source is beautiful. DeepSeek could have kept Engram proprietary. They did not. They open sourced it. Now I can use it. Now everyone can use it. This is how progress works.

Third, I am not special. My ideas are not unique. This is humbling. This is also freeing. I do not need to be the first. I just need to contribute. To try. To build something even if it is built on others' work.

Sonnet Training Update

Sonnet is at 15 percent now. It has been running for days. It will finish eventually. Sonnet-2 will use Engram. It will have external memory. It will still probably give fish answers to math questions. But it will have external memory.

Opus is still a dream. Six hundred million parameters. Forty days of training. With Engram it might actually remember things. That would be new. That would be progress.

Final Thoughts

DeepSeek beat me to my own idea. It hurts. It also frees me. I do not need to build EMM anymore. Engram exists. It is open. I can use it. I will use it.

Thank you DeepSeek. For the idea. For the code. For the lesson in shipping. And for the slight existential crisis. I needed it. Probably.

TMLM-Sonnet-2 will have Engram. TMLM-Opus will have Engram. My tiny models will have external memory. They will still be tiny. They will still be confused. But they will have memory. That counts for something.